Using SoftFileLock on NFS makes it safe for multiple processes on multiple nodes to handle data resources (download and decompress).
The commonly used resource management interfaces are uniformly exposed in llm_web_kit/model/resource_utils/init.py , so there is no need to call them one by one from each file.

Test on Spark4 with 1000 partition:

• update_language_by_str run with an empty cache folder(have to download resource first) with total 546＼*1000 QPS (peak parallelism about 350) and rerun(resource already processed) with 725＼*1000 QPS
• update_political_by_str run with an empty cache folder(have to download resource and unzip files first) with 222＼*1000 QPS (peak parallelism about 100) total and rerun(resource already processed) with 261＼*1000 QPS